feat(pruner): Implement `full` and `bridge` node pruning #3150

renaynay · 2024-01-31T15:45:31Z

This PR introduces full and bridge node pruning via the --experimental-pruning flag. Support is included for nodes that start from scratch with pruning enabled and also for archival (nodes retaining all historical blocks) that enable the --experimental-pruning flag.

Note that this PR does not support the conversion of a pruned node into an archival one explicitly (it would not support re-syncing deleted blocks).

With pruning enabled, full and bridge nodes' block stores can be expected not to exceed ~4TB (as the upper bound).

In follow-up PRs (hardening), the following features can be expected:

discovery for archival nodes for archival sync
inverted_index / light node pruning
include more metrics for errors

TODO:

clean up some TODOs
fix one flakey unit test
change values back to the actual (GC cycle, sampling window, pruning window, etc).
figure out whether to store error in pruner checkpoint
fix issue with pruning genesis block via findPruneableHeaders
metrics for failed prunes
set a sane default for max pruneable / consider removing MaxPruneablePerGC as now context timeouts are on a per block basis
dedup findPruneableHeader test utility
badger dep

codecov-commenter · 2024-02-22T12:25:57Z

Codecov Report

Attention: Patch coverage is 60.08584% with 93 lines in your changes are missing coverage. Please review.

Project coverage is 52.89%. Comparing base (1809680) to head (5918f6d).
Report is 6 commits behind head on main.

Files	Patch %	Lines
pruner/metrics.go	9.09%	30 Missing ⚠️
pruner/service.go	73.80%	18 Missing and 4 partials ⚠️
cmd/util.go	0.00%	14 Missing ⚠️
pruner/checkpoint.go	71.87%	5 Missing and 4 partials ⚠️
pruner/finder.go	83.78%	3 Missing and 3 partials ⚠️
header/headertest/testing.go	72.22%	4 Missing and 1 partial ⚠️
pruner/params.go	54.54%	5 Missing ⚠️
cmd/node.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3150      +/-   ##
==========================================
+ Coverage   51.92%   52.89%   +0.97%     
==========================================
  Files         178      183       +5     
  Lines       11316     9539    -1777     
==========================================
- Hits         5876     5046     -830     
+ Misses       4942     3980     -962     
- Partials      498      513      +15

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

renaynay · 2024-02-28T15:16:38Z

TODO @renaynay:

add some protection in finder.go to prevent

2024-02-28T16:10:59.333+0100	ERROR	pruner/service	pruner/service.go:129	failed to find prune-able blocks	{"error": "header/store: invalid range(26625,26624)"}

(can sometimes happen but is non-critical error, just ugly)

walldiss

Most importantly all time calculations needs to take care of local machine zone, by using time.Now().UTC()

cmd/util.go

nodebuilder/pruner/flags.go

header/headertest/testing.go

nodebuilder/pruner/config.go

nodebuilder/pruner/module.go

pruner/finder.go

pruner/service.go

pruner/params.go

Wondertan · 2024-03-11T12:50:01Z

@renaynay, there is a little conflict fyi

Wondertan

The code is clean and well-structured. It wasn't hard to digest. I didn't find anything critical, and my review is primarily little nits I discovered and questions. There is one general comment about godocs for methods that need to be improved. Otherwise,

pruner/metrics.go

pruner/finder.go

pruner/service_test.go

pruner/service.go

pruner/finder.go

pruner/metrics.go

pruner/params.go

pruner/service.go

go.mod

nodebuilder/pruner/config.go

…s unit tests

…r handling

…bridge

…k time in header test

…increase default max pruneable

…flag

…text deadline should be per block

…her tests deterministic and test one more case for retries

…to proper values, sane MaxPruneable values, return gc cycle to every 5 min

…bit more generous, fix log

…ime, simplify ParseFlags

…ight

…opt,

renaynay added area:storage kind:feat Attached to feature PRs labels Jan 31, 2024

renaynay self-assigned this Jan 31, 2024

renaynay force-pushed the beta.5 branch from a53cdc1 to 26f63ed Compare February 5, 2024 13:12

renaynay mentioned this pull request Feb 15, 2024

feat(p2p/discovery)!: Implement archival discovery + syncing of historic blocks and blobs #3188

Merged

6 tasks

renaynay force-pushed the beta.5 branch 2 times, most recently from 146265a to 1f67976 Compare February 22, 2024 09:54

renaynay marked this pull request as ready for review February 22, 2024 12:31

renaynay requested review from Wondertan, vgonkivs, distractedm1nd, walldiss and ramin as code owners February 22, 2024 12:31

walldiss requested changes Mar 11, 2024

View reviewed changes

renaynay force-pushed the beta.5 branch from babbb62 to fe877c6 Compare March 18, 2024 08:52

renaynay requested a review from walldiss March 19, 2024 08:57

Wondertan reviewed Mar 21, 2024

View reviewed changes

celestiaorg deleted a comment from abstraking Mar 26, 2024

walldiss reviewed Mar 27, 2024

View reviewed changes

go.mod Outdated Show resolved Hide resolved

nodebuilder/pruner/config.go Show resolved Hide resolved

renaynay and others added 8 commits March 27, 2024 15:59

Initial implmenetation of the full pruner

953a407

feat: initial untested findPruneableHeaders impl, findPruneableHeader…

6ee4305

…s unit tests

feat: full node pruner

3b74604

refactor(pruner): making Prune signature non-variadic for easier erro…

1ac1ea9

…r handling

fix(pruner/full): short-circuiting on Prune

89621d9

test(pruner): Fixing expected value in TestService

9b1bcee

feat(pruner): adding metrics

a8fcab1

test: setting availability window to 1 day and adding full pruner to …

3c9500b

…bridge

renaynay added 24 commits March 27, 2024 15:59

test(pruner): Test the pruner service, includes commit to enable bloc…

7b69de7

…k time in header test

feat(pruner): Implement retry functionality

422a949

feat(pruner): make max pruneable configurable and shorten gc cycle + …

cb214e3

…increase default max pruneable

go mod tidy

800d00b

feat(nodebuilder/pruner): Enable pruning with --experimental-pruning …

24ba17b

…flag

chore(nodebuilder/pruner): remove stubbed out code | fix(pruner): con…

a9218e9

…text deadline should be per block

fix(pruner): Metrics callback unify

06dba21

fix | test(pruner): Make TestService more deterministic, Make some ot…

2e9bf7f

…her tests deterministic and test one more case for retries

chore(pruner): remove TODOs

6113788

fix(pruner): findPruneable also prunes genesis block, return windows …

811cd73

…to proper values, sane MaxPruneable values, return gc cycle to every 5 min

fix(pruner): Remove maxPruneablePerGC and introduce maxHeadersPerLoop

5bfc7be

fix(pruner/full): window should be type AvailabilityWindow

f7a183d

fix(pruner): Do not omit FailedHeaders if not set

14fbbc6

chore(pruner): extend prune ctx deadline to 5 sec instead of 1, be a …

32a52d0

…bit more generous, fix log

fix(pruner): checkpoint failed map doesnt store error, UTC used for t…

22c1c17

…ime, simplify ParseFlags

fix(pruner): remove lastPruned header from checkpoint and just use he…

d817db6

…ight

fix(pruner): only log actual errors from findPruneable

80feb82

chore(pruner): idiomatic defer close chan

b7cb726

feat(pruner): change finder algo to use blockTime, remove disabledGC …

3e8b65f

…opt,

fix(pruner): Reduce maxHeadersPerLoop to 512

eafab98

chore: lint

32b8f91

chore(pruner): fix comments from hlib

7e8fd9c

fix(pruner): just short-circuit when there is nothing left to prune

ba66240

fix(pruner): unregister callbck from meter on stop

d5a9784

renaynay force-pushed the beta.5 branch from 149383c to d5a9784 Compare March 27, 2024 15:00

renaynay requested a review from walldiss March 27, 2024 17:55

Wondertan approved these changes Mar 27, 2024

View reviewed changes

walldiss approved these changes Mar 28, 2024

View reviewed changes

renaynay merged commit 02030bf into celestiaorg:main Mar 28, 2024
27 checks passed

renaynay deleted the beta.5 branch March 28, 2024 14:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(pruner): Implement `full` and `bridge` node pruning #3150

feat(pruner): Implement `full` and `bridge` node pruning #3150

renaynay commented Jan 31, 2024 •

edited

Loading

codecov-commenter commented Feb 22, 2024 •

edited

Loading

renaynay commented Feb 28, 2024

walldiss left a comment

Wondertan commented Mar 11, 2024 •

edited

Loading

Wondertan left a comment

feat(pruner): Implement full and bridge node pruning #3150

feat(pruner): Implement full and bridge node pruning #3150

Conversation

renaynay commented Jan 31, 2024 • edited Loading

codecov-commenter commented Feb 22, 2024 • edited Loading

Codecov Report

renaynay commented Feb 28, 2024

walldiss left a comment

Choose a reason for hiding this comment

Wondertan commented Mar 11, 2024 • edited Loading

Wondertan left a comment

Choose a reason for hiding this comment

feat(pruner): Implement `full` and `bridge` node pruning #3150

feat(pruner): Implement `full` and `bridge` node pruning #3150

renaynay commented Jan 31, 2024 •

edited

Loading

codecov-commenter commented Feb 22, 2024 •

edited

Loading

Wondertan commented Mar 11, 2024 •

edited

Loading